Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow building of the ufs-weather-model, WW3 pre/post execs for GFS, GEFS, SFS in the same clone of global-workflow #3098

Open
wants to merge 36 commits into
base: develop
Choose a base branch
from

Conversation

aerorahul
Copy link
Contributor

@aerorahul aerorahul commented Nov 14, 2024

Description

GFS and GEFS (and now SFS) uses different compile time options for the UFS-weather-model. For the purposes of CI testing, a multi-build pipeline under Jenkins was created by @TerrenceMcGuinness-NOAA. This served well, until now. With the inclusion of SFS, a third variety of the model is being built. Under the multi-build pipeline paradigm, a second (or third) clone and build of the global-workflow is required. This adds cloning and compilation time of the global-workflow.

This PR allows compiling the ufs-weather-model in a single clone of the global-workflow. The compiled executables based on the options for GFS, GEFS, and SFS results in a model executable as gfs_model.x, gefs_model.x, and sfs_model.x. The forecast script uses the right executable.

This PR also differentiates the WW3 pre/post executables based on gfs_ or gefs_. SFS variants need to be introduced, when NET=sfs is added.
In the process of updating the WW3 pre/post executable names, it was discovered (via grep) the following are not used:

  • ww3_prep
  • ww3_outf
  • ww3_ounf
  • ww3_ounp
    These need to be confirmed by running the workflow to ensure they are indeed not used at runtime.

This PR also updates the Jenkinsfile to use the multi-build from the single location.

Type of change

  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

This PR does not update any submodules.

How has this been tested?

In progress

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

sorc/build_ww3prepost.sh Fixed Show fixed Hide fixed
scripts/exgfs_wave_post_pnt.sh Fixed Show fixed Hide fixed
ush/wave_grib2_sbs.sh Fixed Show fixed Hide fixed
ush/wave_grib2_sbs.sh Fixed Show fixed Hide fixed
ush/wave_grib2_sbs.sh Fixed Show fixed Hide fixed
ush/wave_grid_interp_sbs.sh Fixed Show fixed Hide fixed
ush/wave_grid_moddef.sh Fixed Show fixed Hide fixed
ush/wave_prnc_cur.sh Fixed Show fixed Hide fixed
@aerorahul aerorahul changed the title Allow building of the ufs-weather-model for GFS, GEFS, SFS in the same clone of global-workflow Allow building of the ufs-weather-model, WW3 pre/post execs for GFS, GEFS, SFS in the same clone of global-workflow Dec 5, 2024
@aerorahul
Copy link
Contributor Author

I am going to mark this as ready for review to gather feedback on the code changes before I commence extensive testing.

sorc/build_ww3prepost.sh Fixed Show fixed Hide fixed
sorc/build_ww3prepost.sh Show resolved Hide resolved
ush/wave_grid_interp_sbs.sh Outdated Show resolved Hide resolved
sorc/build_ww3prepost.sh Show resolved Hide resolved
@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed label Dec 12, 2024
@WalterKolczynski-NOAA
Copy link
Contributor

WalterKolczynski-NOAA commented Dec 12, 2024

C48_S2SW is still failing, now at wavepostsbs when it tries to execute gfs_ww3_grib.x. Error code 41

Digging in, I find a couple different things. The ww3_grib error seems to come an inability to open a file (output_20210323120000/glo_200_grib/grib2_global_000.out):

 Additional GRIB parameters : 
 -----------------------------------------------------
      Run time           : 2021/03/23 12:00:00 UTC
      GRIB center ID     :    7
      GRIB gen. proc. ID :   11
      GRIB grid ID       :  255
      GRIB GDS parameter :    0

 *** WAVEWATCH III ERROR IN W3IOGO : 
     ERROR IN OPENING FILE
     IOSTAT =   29

There is a zero-size gribfile and a dangling symlink for out_grd.ww3

There is also a failure in wave_grib_interp_sbs because it is trying to write to fix, which I think is supposed to be the source of the out_grd.ww3 target:

+ wave_grid_interp_sbs.sh[147](glo_200): '[' no = no ']'
+ wave_grid_interp_sbs.sh[149](glo_200): cp -f ./WHTGRIDINT.bin /lfs/h2/emc/stmp/walter.kolczynski/RUNDIRS/C48_S2SW_3098/gfs.2021032312/wavepostsbs.181086/ww3_gint.WHTGRIDINT.bi
n.glo_200
+ wave_grid_interp_sbs.sh[150](glo_200): cp -f ./WHTGRIDINT.bin /lfs/h2/emc/ptmp/walter.kolczynski/PR/PR_3098/global-workflow/fix/wave/ww3_gint.WHTGRIDINT.bin.glo_200
cp: cannot create regular file '/lfs/h2/emc/ptmp/walter.kolczynski/PR/PR_3098/global-workflow/fix/wave/ww3_gint.WHTGRIDINT.bin.glo_200': Permission denied
+ wave_grid_interp_sbs.sh[1](glo_200): postamble wave_grid_interp_sbs.sh 1733980516 1
+ preamble.sh[70](glo_200): set +x
End wave_grid_interp_sbs.sh at 05:15:28 with error code 1 (time elapsed: 00:00:12)

This doesn't immediately fail, but continues until the ww3_grib failure later.

wave_grid_interp_sbs.sh definitely shouldn't be writing to fix/wave, but I don't understand why this is only a problem now.

The logs don't all get piped to STDOUT, so you have to root around in the DATA directory to find them (here's one: /lfs/h2/emc/stmp/walter.kolczynski/RUNDIRS/C48_S2SW_3098/gfs.2021032312/wavepostsbs.181086).

@WalterKolczynski-NOAA
Copy link
Contributor

I'll also note the C48_S2SWA_gefs case completes successfully.

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

This fail is from PR 307 in my forked repo for testing this build using the updated CI pipline:

running the C48mx500_hybAOWCDA FAILED on Hercules in Build# 6 with error logs:

/work2/noaa/global/CI/HERCULES/307/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_22a4e28b/logs/2021032500/gdas_marineanlletkf.log

Follow link here to view the contents of the above file(s): (link)

All the other CI tests are all languishing in the Priority queues on Hercules

…s another bug that needs evaluation on the purpose of the logic. A FIXME tag has been added
@aerorahul
Copy link
Contributor Author

@guillaumevernieres

This fail is from PR 307 in my forked repo for testing this build using the updated CI pipline:

running the C48mx500_hybAOWCDA FAILED on Hercules in Build# 6 with error logs:

/work2/noaa/global/CI/HERCULES/307/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_22a4e28b/logs/2021032500/gdas_marineanlletkf.log

Follow link here to view the contents of the above file(s): (link)

All the other CI tests are all languishing in the Priority queues on Hercules

@TerrenceMcGuinness-NOAA
I am not sure why this is the case. It seems like files are missing. I will wait for 3149 to finish running on Hercules to confirm it ran this test successfully in that PR.

Copy link
Collaborator

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The updates to the Jenkinsfile pipeline script look good. The try/catch on the scm checkout looks strong too. An initial test of the pipeline worked fine in a forked branch in the development project. When CM (@WalterKolczynski-NOAA) is ready we should launch a CI test directly in this PR using multiple labels.

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Dec 12, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Dec 12, 2024
@WalterKolczynski-NOAA
Copy link
Contributor

Will run CI on Hercules as well once the PRs already running there have completed.

@emcbot emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Dec 12, 2024
@emcbot
Copy link

emcbot commented Dec 12, 2024

CI Passed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/3098


Experiment C48_ATM_50794488 Completed 2 Cycles: *SUCCESS* at Thu Dec 12 19:37:04 UTC 2024
Experiment C48mx500_3DVarAOWCDA_50794488 Completed 2 Cycles: *SUCCESS* at Thu Dec 12 19:37:04 UTC 2024
Experiment C48mx500_hybAOWCDA_50794488 Completed 2 Cycles: *SUCCESS* at Thu Dec 12 19:43:22 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_50794488 Completed 1 Cycles: *SUCCESS* at Thu Dec 12 19:50:06 UTC 2024
Experiment C96C48_hybatmaerosnowDA_50794488 Completed 3 Cycles: *SUCCESS* at Thu Dec 12 20:56:27 UTC 2024
Experiment C96_atm3DVar_50794488 Completed 3 Cycles: *SUCCESS* at Thu Dec 12 21:02:22 UTC 2024
Experiment C96C48_hybatmDA_50794488 Completed 3 Cycles: *SUCCESS* at Thu Dec 12 21:02:33 UTC 2024
Experiment C96C48_ufs_hybatmDA_50794488 Completed 3 Cycles: *SUCCESS* at Thu Dec 12 21:32:57 UTC 2024
Experiment C48_S2SW_50794488 Completed 2 Cycles: *SUCCESS* at Thu Dec 12 21:51:52 UTC 2024
Experiment C48_S2SWA_gefs_50794488 Completed 1 Cycles: *SUCCESS* at Thu Dec 12 22:22:51 UTC 2024

@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Dec 13, 2024
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Dec 13, 2024
@emcbot emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants